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ABSTRACT 

A study analyzed the prior knowledge state of two 
university populations: students at the Open University (OU) of the 
Netherlands and students at the University of Limburg (UL) . Analysis 
of the prior knowledge state was based on an extensive analysis of 
the literature in relation to theories, models, and practice-based 
strategies about the structure of knowledge. A set of four 
dimensions — cognitive psychological, educational psychological, 
psychometrical , and content based — that help to construct knowledge 
profiles was defined. The dimensions were used to analyze the 
knowledge profiles of the two populations. A knowledge state test 
that covered the whole domain of economics in the course "Economics 
and Money" was administered to 626 students; 91 UL students were 
randomly selected to be compared with 91 OU students. The overall 
economics score was not significantly different between the two 
populations. This score was elaborated upon by grouping items along 
knowledge profile dimensions. After outlining the theoretical base of 
these dimensions, researchers focused on the application of these 
dimensions to compare the knowledge profiles of the two populations. 
Up to 7 of the 10 knowledge profile dimensions proved to be relevant. 
The content and epistemological dimensions were especially helpful to 
describe differences. (Contains 28 references.) (YLB) 
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OTIC RESEARCH REPORTS. 

The Open University is responsible for developing And offering open, higher distance education in which special 
attention is paid to innovations in educational technology. The research in this field is concentrated in "OTIC", that is 
the Centre for Educational Technological Innovations (Onderwijs Technologist Innovatie Centrum). 
OTIC is also engaged in running projects for other institutions. Here the Centre makes use of OTIC's knowledge and 
experience acquired in research and development. 

The series of OTIC Research Reports consists of publications of the OTIC research projects and aims mainly at an 
audience of fellow researchers. 



THE RESEARCH PROJECT ON EVALUATION AND TEST-FUNCTIONALITIES 

The project "Evaluation and Test-functionalities" focuses on the problems caused by the wide diversity of students and 
the problems with individual and flexible karnmg-processcs of these students. The project leads to an integration of 

- the results of the project "Prior Knowledge"; 

- the developments of the "Computer Assisted Testing"-project; 

- the developments of "Adaptive Testing" and the IRT-appIications (Item-Responds Theory); and 

- the experience of the Open University with the development and use of TSS (Test service systems). 

The main objectives are: (1) to get a discernment of the test and evaluation problems in the open-learning system; (2) 
the generation of the guide-lines, specifications, and technological instruments concerned with the use of prior 
knowledge and experience, flexible testing and the supervision on students during the learning process, and (3) the 
development of instruments which can be useful in solving the given teaching problems. 
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Introduction 



When studying at university level, students have - in the Dutch context - the option to choose from a variety 
of contexts and educational approaches. A relatively new development in this perspective is the provision of 
open and distance university education by the Dutch Open university. One can question whether students, 
opting for this study context are different from students studying at regular universities. A possible 
difference can be reflected in the prior knowledge state of these students. 

The analysis of the quality and impact of the prior knowledge state has been the maji r fccus in a large part 
of our earlier research. In analyzing the prior knowledge state, we have already focu- d on the structure of 
the prior knowledge state along a content dimension. 

In the theoretical part of this text, we discuss - in short - a distinct approach towards the analysis of the 
prior knowledge state'. This approach is based on an extensive analysis of the literature in relation to 
theories, models and practice-based strategies about the "structure of knowledge". This base ir exploited to 
define a set of "dimensions" that are helpful to construct "knowledge profiles". Four types oi dimensions 
are illustrated : cognitive psychological dimensions, educational-psychological dimensions, psychometrical 
dimensions and content-based dimensions. 

In the empirical part of this text, the dimensions are used to analyze the knowledge profiles of two 
university populations : students studying a the Open university (Ou) of the Netherlands and students 
studying at the University of Limburg (RL). The results of this analysis are not only important to detect 
specific differences in the mastery of components of the prior knowledge state between both student 
populations, but might also be helpful to provide further evidence about the validity of the theoretical 
knowledge profile dimensions. 



2 Theoretical background 
2. 1 The Structure of Knowledge 

From an instructional-psychological point of view, the structure-of-knowledge problem should be 
investigated in order to find out more efficient ways for using instructional technology. Our search for 
means to handle the prior knowledge state showed that one should take account different components of 
the prior knowledge state. The concept of "components" refers towards a structure in the knowledge base 
of the learner. 

Our earlier research was helpful to detect such components of the prior knowledge state along the content 
dimension. But it was also suggested that the differentiation of components of the prior knowledge state 
along other dimensions is needed to be helpful to interfere and diagnose educational practice (Dochv and 
Valcke, 1991; Dochy and Valcke, 1991b). 

The issue of the "structure of knowledge" has been debated from a variety of theoretical points of view : 
cognitive psychology, epistemology, philosophy, etc. At the more pragmatic level, the issue has also been 
of prime importance in applied sciences like instructional psychology, curriculum development theories and 
psychometry. Disciplines like cognitive psychology, educational psychology, artificial intelligence, etc. - 
have - from their points of view - highlighted the "structure of knowledge" resulting in a puzzling variety of 
approaches, focuses, models, theories, research attempts, ... . A representative sample of authors 
comprises e.g. Ausubel (1968), de Groot (1946), Mayer (1979), Reigeluth and Stein (1983). 

It should be noted that our primary focus in using these theories originates from an information processing 
view on learning (Sternberg, 1985a & 1985b). The main reason for this is that we stress a dynamic 
approach towards the structure (knowledge acquisition) of knowledge, which is in particular advocated in 



' A more eUbortted vertion of the theoretic*! bete of the knowledge profile dimeneione cen be found in : Dochy & Velcke (1991e) 
Vehdetton of Knowledge Profile Dimeneione : Looking for empiricel Evidence. OTIC Reeeerch report 33 . Heerlen : Ou-OTIC. 
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this view. If we summarize the variety of approaches, four main types of dimensions to structure 
knowledge can be conceptualized : 



Content related dimensions 
Educational dimension! 
EpUtemological dimensions 
Item characteristic dimensions 



As discussed in another publication (Dochy and Valcke, 1991), some of these dimension are of a 
hierarchical nature. 

2.2 Knowledge Profiles 

As such, the concept of "knowledge profiles* is not found in literature. Only 'student profiles* (Wolf, et. 
al., 1991) and 'cognitive profile* (Letteri et. al., 1980) have some similarity in meaning. This is certainly 
the case for the studies by Letteri et. al. (1980, 1982). The concept 'profile* is derived from the practice, 
common in educational research, of plotting as a graph or profile the scores of a person as raw scores or as 
standardized scores (Keeves, 1988). In analyzing research findings, comparisons are made between persons 
or groups in terms of a set of measurements on specific related aspects. For each person or group a profile 
is obtained on a set of parameters. The comparison between profiles of persons is known by the generic 
term 'profile analysis*. 

Figure 1 shows the relationship between some key concepts. A "dimension* is used to construct a 
knowledge profile. Each dimension represents an approach towards the structure of knowledge. The 
structure components are named "parameters". 





i 


profile 


dimension 

parameter 



Figure 1: Example of a profile 



From an instructional psychological point of view, knowledge profiles can give practical indications of 
student achievement and learning in order to direct the learning process. In a recent overview of student 
assessment, Wolf et. al. (1991) advocate this approach. According to these authors, there is a need for new 
educational psychometrics capable of answering the much changed question of educational achievement. 
These changes are the new premises, the multiple paths towards the prior knowledge state, more 
developmental oriented assessments and the ascertainment that students enter school with widely varying 
backgrounds. In our terms, we take account of these changes by trying to identify multiple components of 
the prior knowledge state by implementing prior knowledge state tests and by intending to use these tests as 
progress testa administered several times a year. In this cor text it is necessary to come to an agreement on 
the relevant parameters to describe student performance anu it is critical to develop ways of looking at 
'student profiles*: " unless we develop these kinds of differentiated portraits of student performance within a 
domain, it is difficult to envision student assessment ever informing, rather than merely measuring, the 
educational process" (Wolf, et. al., 1991). 
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2.3 Overview of Knowledge Profile Dimensions 

Only those dimensions/parimeters are reviewed that have been retained after their discussion and analysis in 
our earlier publication "Validation of Knowledge Profile Dimensions : Looking for empirical Evidence". 
If dimensions are based on a model or theory, only short details will be reported. 

The first dimensions are classified according to common models of economics. Other dimensions are based 
on theories of knowledge representation, knowledge structure, learning theories, text representation models 
and psychometric theory. 

2.3.1 Content Dimensions 

Economics aflafaffllfflg tinm g i nn 

"Content" is a commonly used dimension to categorize domain knowledge. Classification based on the 
parameter 'subdomsins* refers to the subdivision of the economics-domain into "subject nntter blocks" that 
are standard within the science of economics. Our dimension structure is e.g. based on the curriculum 
structure of the University of Maastricht and reflects 9 parameters : 



Curriculum level dimension 

Some parts of the content of a science are supposed to be mastered by the students at certain moments 
during their study. These moments are called the curriculum levels (first and second year). These levels 
are subsequent, but too broad to be supposed hierarchical. 



Curriculum accent dimension 

Within economics it is common to differentiate between two main streams, representing a different accent, 
i.e. general economics and business administration on the one hand and quantitative economics on the other 
hand. 




1. Reporting 

2. Financing 

3. Organization 

4. Marketing 

5. Macro-economics 

6. Mfcio-ecooomics 

7. Public finances 

S. International economic affairs 
9. Behavioural and social sciences 




1 . General economics and business administration 

2. Quantitative economics 
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2.3.2 Cognitive Psychological Dimensions 

Knowledge representation, as used in schema theories (Dochy and Bouwens, 1990), takes certain 
propositions or nodes as a starting point. A proposition is the smallest unit that can be qualified as true or 
false in a statement. According to most schema theories there are five lands of nodes : Physical State (PS, 
statement that refers to an ongoing state in the physical or social world), Physical Event (PE, statement that 
refers to a state change in the physical or social world), Internal State (IS, statement that refers to an 
ongoing state of knowledge, attitude, or belief in a character), Internal Event (IE, refers to a state change in 
knowledge, attitude or belief in a character), Goal (G, statement that refers to an achieved or unachieved 
state that a person wants) and Style (S, statement that refers to details about the style or manner in which an 
action or event occurred. 



The "Node Relation" dimension is based on characterises of the interrelations between propositions, called 
node relation or arc parameters: Reason (R, a Goal node is a reason for another Goal node), Initiate (I, a 
State or Event initiates another Goal node), Consequence (0, a State, Event or Goal node that has the 
consequence of another State or Event node), Manner (M, an Event or Goal node occurs with some style), 
Property (P, a person, object or entity has some property that is a State node) (see also Dochy and 
Bouwens, 1990). These arc parameters are not of a hierarchical nature. 

2.3.3 Educational-Psychological Dimensions 

The theoretical base of these two dimensions - i.e. behavioural and content dimension - is found in 
Component Display Theory (CDT, Merrill, 1983), Taxonomy theories (De Block, 1986 and Bloom, 1956) 
and Gagn6's theoretical classification (198S). 

Behavioural dimension 

The known distinction between declarative and procedural knowledge is further opeiationalised at this stage 
into the parameters *to know, to understand, and to apply*. These parameters axe also perceived as 
equivalent to the concepts 'recognition, reproduction and production'. Items can be classified as measuring 
the appreciation, the recognition and the reproduction of information (declarative) or measuring production 
or applications (interpretative, convergent, divergent or evaluative production = procedural) (Keeves, 1988). 





IS - PE 
PE-PE 
IE - PE 
G -PE 
PS-PS 
IS - PS 
PE-PS 
IE- PS 
G - PS 
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The three parameters do also correspond with taxonomic levels proposed by several educationalists as 
Bloom, Guilford, De Cbrte and De Block (cf. Keeves, 1988). Most researchers agree that these parameters 
are hierarchical in nature. 



l.Know 


1. Declarative 


2. Uadentaod 




1 3- Apply 


2. Procedural 



Content dimension 



Along the content dimension we differentiate five parameters : facts, concepts, relations, structures and 
methods. This is in accordance with e.g. the work of Guilford when he refers to product parameters 
(Keeves, 1988). These parameters are widely accepted as being hierarchical (Keeves, 1988). 



1. Facte 

2. Concept! 

3. ReUtioos 

4. Structure* 

5. Method* 



Epistemological diffiSflSjoa 

Based on the levels of knowledge representation of Brachman (1985), five parameters can be differentiated 
along a typical dimension. These parameters car also be considered as the most appropriate combinations 
of behaviour- and content levels, as clarified between brackets : knowledge identification (identifying facts 
and concepts), knowledge conceptualisation (insight in concepts), epistemological analysis (to know and 
understand, relations and structures), logical analysis (to know and understand methods), implemenUtional 
analysis (application of methods). These levels are considered as hierarchical since they are a combination 
of the hierarchical behavioural and content level. 



1 . Knowledge identification 

2. Knowledge conceptualisation 

3. Epittemotogical analysis 

4. Logical analysts 

5. ImplemenUtional analysis 



2.3.4 Item Characteristics Dimensions 
Number of propositions dimension 

A proposition is the smallest unit that can stand as a separate assertion which can be judged as true or false 
In schema theories (Dochy and Bouwens, 1990), propositions or nodes have a core function in the structure* 
of schemata. It is assumed that the amount of propositions determines the degree of structure needed to 
answer the item correctly. Three parameters have been identified in relation to this dimension : 



1 . < 5 propositions 

2. > 4 < 10 propositions 

3. > 9 propositions 
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The "stem" of an item is the general information which is given and which must not be evaluated. This 
correct information precedes the questions for which this information should be taken into account. A stem 
can be connected to one or more subsequent questions. Therefore, the spatial and logical distance between 
the general information part of an item and the question part is larger than for simple items without a stem. 



1. hern* with a item 

2. Kerns without a stem 



Representation level dimension 

Following the classification used in the research of Boekaerts (1979), i.e. visual, verbal and symbolic 
representation, we distinguish four parameters along this dimension. These parameters are also closely 
related to the four content levels of Guilford's structure of intellect model: figural, symbolic, semantic (the 
verbal factor) and behavioural (nonverbal information) and the Twyman (198S) categories : verbal, pictorial 
and schematic. 



Test-items are always based on textual information representation, but can be enhanced, enriched or 
documented with information of an other representation category : 



1 . Textual-graphical 

2. Textual 

3. Schematic 

4. Textual-symbolic 



ical I 
I 



2.4 A rationale for Differences between University Populations 



As explained in the introduction students have - in the Dutch context - the option to choose for a variety of 
university contexts and educational approaches at university level (e.g. problem centred approach, 
experiential learning, distance education, etc). A relatively new development (since 198S) in this 
perspective is the provision of "open and distance university education" by the Dutch Open university (Ou). 

The question can be put forward whether this new university setting is just another higher education 
institution enriching the variety of already existing provisions or whether the Open university answers the 
need of (a) specific student population(s); e.g. second chance, older students, female students, handicapped 
people, foreign students, post-university students ? A way to look for answers to this question is to 
analyze - by interviews, questionnaires, etc. - demographic variables of the actual student population of the 
Open university. 

Another approach goes beyond these surface variables and analyzes in more detail the prior knowledge state 
of the students opting for the Ou. The logical research question, which results from this approach is 
whether the prior knowledge state of the students, opting for this study context is different from students 
studying at regular universities. 

The analysis of the prior knowledge state has been the major focus in a variety of research projects at the 
Op^n university (Centre for Educational Technology and Innovation). Among the important findings of 
these studies, we mention : 

Personal and contextual variables are no good "indicators" of the prior knowledge state. 

Subjective ratings of the prior knowledge state do not reflect the real prior knowledge state 

level. 

Components of the prior knowledge state can be identified along the content dimension 
(e.g. mathematics, optimal requisite knowledge, etc.)* 
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As explained in the former parts of this text, the development of "knowledge profiles" has a large potential 
to describe in more detail "components" of the prior knowledge state. In the further part of this text, we 
will try to apply this new approach to the central question of this research : Is studying at a different 
University reflected in differences in the prior knowledge state ? 



3 Research design 

3.1 Hypotheses 

Taking into account the theoretical base of the present study, the following main hypothesis can be stated : 

"Ou-students and RL-students are different in Urns of the parameters 
along a variety of knowledge profile dimensions". 

Since up to 10 profile dimensions will be used, the main hypothesis can be split up into a set of 10 
subhypotheses : 

• Ou-students and RL-students have a different economics subdomains knowledge profile. 

- Ou-students and RL-students have a different curriculum knowledge profile. 

- Ou-students and RL-students have a different curriculum accent knowledge profile. 

- Ou-students and RL-students have a different node relation knowledge profile. 

- Ou-students and RL-students have a different behavioural knowledge profile. 

- Ou-students and RL-students have a different content knowledge profile. 

- Ou-students and RL-students have a different epistemological knowledge profile. 

- Ou-students and RL-students have a different representation level knowledge profile. 

- Ou-students and RL-students have a different number of propositions knowledge profile. 

- Ou-students and RL-students have a different information level knowledge profile. 

3.2 Research Instruments 

As described above, a domain specific knowledge state test was administered to the research population. 
This test consists of 154 items. The test covers the whole domain of economics to be studied at university 
level in relation to the courses "Economics & Money". 

This test consists of multiple-choice questions which can be answered with true/false or ?. The ?-alteraative 
is taken as a third alternative in order to prevent guessing. 

The particular characteristics of this test already suggest that the determination of its psychometric qualities 
might be a problem. There is no problem in relation to validity since the test clearly represents - to a very 
large extent - the domain and has been developed by a team of domain experts. The psychometric quality 
problem is especially in game when determining the reliability of the test. If we calculate the alpha- 
coefficient, the test can be considered as very reliable : a = .9302. 

But this high reliability level is marred by the fact that the test is very long (154 items); thus resulting rather 
easily in a high or-coefficient. Moreover, the test is not homogeneous (cf. the different profile dimensions 
specified above) what implies that basic assumptions to calculate the a-coefficient have been violated. 



ERLC 



13 



Knowledge profiles of OU and RL students 



TabU 1 : 

a-coefficicnts for the subdomains and curriculum accent dimension 



DsflMSMsos^pajraaseteri 


0 


• w 




Repotting 


.5739 


18 




Financing 


.6449 


18 




Organization 
Marketing 


.6922 


18 




.6292 


18 




Macro-economies 


.7069 


25 




Micro-economics 


.7420 


25 


.631 J 


Public Finances 


.5101 


11 




International Economic affairs 


.5543 


11 




H Behavioural & Social Sciences 


.62*7 


10 




1 General economics & Business Administration 


.9270 


135 


.686 | 


| Quantitative economics 


.4467 


15 





A solution to this problem might be to check the reliability of subparts of the test, making use of the 
knowledge profile dimensions. Calculation of a was repeated for two of these dimensions (course subtopics 
and curriculum accent), in order to be able to present a mean reliability score. The results of this procedure 
are summarized in table 1. To be able to judge the figures in a better perspective, the number of items each 
subgroup of items consists of is also given. 

Mean a seems to be > .63. This reliability score is - taking into account the restricted number of items in 
certain subparts of the test - acceptable for our research purposes. 

3.3 Research Population and Procedure 

The test was administered to a sample of economics students studying at the Dutch Open university and the 
Maastricht University. This sample consisted of 626 students. 

The domain specific knowledge state test was administered to a sample of Ou-students (N=91) and RL 
students (N«536). To obtain equal sample sizes, a random sample (N=91) was selected from the large 
RL-population of the Maastricht university. 

The raw scores for the test items were recoded in order to gather a maximum of information in relation to 
mastery or non-mastery of the domain specific knowledge. After recoding 2 , a general economics-score for 
the entire test was calculated. 

In a next step, all items were classified along the dimensions discussed in part 2 of this text. The 154 items 
were analyzed - separately - by three researchers. In reviewing the items, the researchers attempted to 
classify each item on each one of the 10 dimensions. An inter-rater reliability was obtained > .8T 3 . If 
there was discussion in relation to the categorization of a specific item along a dimension, discussion 
resulted in a consensus on the final evaluation of the item. 

Grouping the items along the knowledge profile dimensions helped to calculate specific subscores. To ease 
comparison of mean total subscores, the individual subscores have been calculated as % -scores. 



9 
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1 The normal scoring procedure for this test implies that students obtain + 1 when their answer is correct; obtain 0 when they answer with 
? and obtain -1 when their answer is wrong. In the recoding process, scoring for wrong answers was changed into a zero-score. 

* • « p 2S .05 m p £ .01 
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4 Discussion of the research results 
4.1 General Results 

Table 2 on the next page gives an overview of the mean scores and subscoies for the entire test and the 
different regroupings of items along the 10 dimensions. 

The name of each dimension is given in the first column. Next the names of the different parameters along 
the dimensions are recited, with - in the third column - the number of items that have been identified as 
exponents of this parameter. 

A striking fact in the table is the large difference in the number of items that help to calculate the 
parameter-subscores. Some N^-values are even problematic. For instance : the "Reason" parameter along 
the "Node Relation" dimension is represented by only two items, making this parameter less useful and 
weakening the validity of this dimension. This has to be taken into account when interpreting the analysis 
results. re / 

The mean % score of the total research sample for each specific parameter is reported in the fourth column. 
The mean % scores for the different parameters in relation to each dimension show striking differences. 
The economics subdomains dimension presents for instance mean % scores varying from 17.97 % to 38.77 
%. This suggests - at a first level - that some dimensions/parameters are helpful to indicate mastery or non- 
mastery of specific components of the prior knowledge state. 

Next to the differences in mean scores, especially the large rvalues draw our attention. These large values 
are the result of the fact that the test measures " the prior knowledge state" of starting university students 
with a wide variety of prior experiences in relation to the topics assessed by the test. 
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Table 2 : 
Overview of general results 



DIMENSION 


PARAMETERS 
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4.2 Profiles of Ou & RL Students : a first Analysis 



50 - 




economics 

Ec. Subtoplc 



Ou Grand mn 223 RL 



Figure 2 : Mean economics scores of Ou-studenls and RL 

Fig. 2 depicts the differences in the mean overall test score of Ou-students and RL. This difference is not 
significant 4 (F=3.747, p^.054). This is consistent with our earlier research findings (Dochy and Valcke, 
1991b). In these research reports the university setting was considered as a context variable, next to other 
personal variables, which was not helpful to predict differences in the prior knowledge state or was not 
helpful to clarify differences in between the student populations. But as also suggested in these earlier 
research reports (Dochy and Valcke, 1991) a more thorough analysis of the overall economics-score for the 
Prior Knowledge State-test can be helpful to reveal - nevertheless - specific (and significant) differences. 

4.2.1 Economics Subdomains Dimension 
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Figure 3 : Economics Subdomains knowledge profile 

The data in fig. 3 reveal clear differences in the mean % scores of Ou- and RL-students for the different 
economics-subtopics. The mean % scores of Ou-students are higher for most subscores, with the exception 
of "macro-economics" and "public finances". 



4 The concept 'significant* in this report refers to 'statistically significant* results. 
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it is therefore interesting to test the significance of these differences in mean % scores 5 . This analysis 
reveals that the differences in mean % -scores of Ou- and RL-students are significant for the subscores 
"reporting", "finance", "organisation", "marketing" "public finance" and "international economics". 

4.2.2 Curriculum Dimension 
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Figure 4 : Curriculum knowledge profile 

As expected, the mean % scores for the level-2 items are lower for both sub-populations. It is normal that 
the prior knowledge state of these advanced level questions remains rather restricted. 
For level- 1 items there's no significant difference between Ou-students and RL students (F=.126, 
Pf=.723). Ou-students score higher than RL-students on level-2 items and this difference is statistically 
significant (F= 18.347, p^.OOO). 

4.2.3 Curriculum Accent Dimension 
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Figure 5 : Curriculum accent knowledge profile 



| If we analyze the correlation matrix of the mean * acoret for the different economics subtopics, we perceive some high 
intercorrelations. Thi» haa to be taken into account whan comparing the mean % scores of the two student-populations. At first level, 
we neglect these correlations between the scores for the different economics-subtopics, to test the significance of the differences 
between the mean % scores by a univariate F-teet. In part 4.3 of this text, we wilt take the interconelation between the subtopics into 
account when executing a multivariate Fleet. 
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The curriculum accent profile shows that "general economics* mastery is higher than "quantitative 
economics" mastery. Also interesting is the fact that Ou-students have higher mean % scores than RL- 
students for "general" as well as "quantitative economics" mastery, although this difference remains non- 
significant. 

4.2.4 Node Relation Dimension 
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Figure 6 : Node relation knowledge profile 

There is a difference between Ou- and RL-students in the mastery of all node relation categories. With the 
exception of the "relation" node category, Ou students score higher than RL-students. The biggest 
difference between the mean %-scoro of Ou- and RL-students is observed in association with the "initiate" 
category. Only the latter difference is statistically significant (F=33.693, pt= .000). 

4.2.5 Behavioural Dimension 
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Figure 7 : Behavioural knowledge profile 

It is found that there is a difference between Ou- and RL-students in the behavioural level knowledge 
profile. Typically we found that this difference is only manifest at the "apply" level. This difference 
between Ou- and RL-students is significant (F= 17.851, p^.OOO). 
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Figure 8 ; Content knowledge profile 

If we consider the "content level" knowledge profile we can conclude that for some categories there is a 
difference between Ou- and RL-students: Ou-students have a superior master} of "factual" knowledge" 
(F=27.260, PF=.000) t "concepts" (F=5.737 t p^.000) and "skills" (F=21.146, p F =.000). 

4.2.7 Epistemological Dimension 
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Figure 9 : Epistemological knowledge profile 

In the epistemological knowledge profile we see again differences between Ou- and RL-students. With Jhe 
exception of the epistemological analysis category, Ou-students always perform superior. The differences 
between the two groups are significant (Kident (F= 11.314, p F =.001); Kconcept (F=9.400, p F =.003); 
Logical (F=7.302, p F =.008); Implem (F=18.451, p^.OOO)). 
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4.2.8 Number of Propositions Dimension 
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Figure 10 : Number of propositions knowledge profile 

Items with a high number of propositions are more complex than items with a low number of propositions. 
It is typically that Ou-students perform better at all proposition-levels than RL-students. Only the difference 
at the highest level is significant (F=8.135, fr=.005), which means that Ou-students deal better with 
complex items than RL-students. 

4.2.9 Information Level Dimension 
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Figure 11 ; Information level knowledge profile 

Mastery of items without a stem seems to be slightly higher than mastery of items with a stem. This can be 
related to the findings in relation to the number of propositions. Items without a stem are based on a 
smaller amount of propositions and are therefore seemingly more easy to comprehend and to solve. 
Although Ou-students perform better in relation to both types of questions, these differences in mean %- 
scores are not statistically significant. 
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4.2. 10 Representation Level Dimension 
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Figure 12 : Representation level knowledge profile 

The mastery of representation levels suggests that RL-students are more able to solve questions based on 
graphical information. Ou-students seem to master questions, based on textual, schematic and symbolic 
information to a higher extent. With the exception of textual representation, these differences in mean %- 
scores are statistically significant (graphical F=8.135, p^.005, schematic F=5.638, pr=.019, symbolic 
F=8.795, Pk=.003). 

4.2. 1 1 Intermediate conclusions 

Although the overall prior knowledge state mean % score for the economics domain is not significantly 
different between the Ou- and the RL-student population, the knowledge profiles seem to be helpful to 
enlighten obvious differences in the complex composition of components of the prior knowledge state. 
Table 3 gives a summary of the analysis data of the univariate analysis of variance : 
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TabUS, 



• scores of RL and Ou students 
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4.3 



4.3.1 



Profile analysis 

Profile analysis as an extension of multivariate analysis of variance 
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A univariate analysis of variance neglects the intercorrelaUons between the different parameters on the 
profile dimensions. These intercorrrelations are important (although not making the specific variables 
redundant) and can be explained at the theoretical level as illuminated elsewhere (Dochy and Valcke, 
1991a). Therefore, a multivariate analysis of variance is needed to refine our analysis and to look for more 
conclusive information about the differences in the prior knowledge state between Ou-students and RL- 
studente. A multivariate analysis can take these intercorrelations into account. Profile analysis is an 
extension of multivariate analysis and is especially appropriate and helpful to evaluate the parameter 
structure in relation to each profile dimension when comparing populations. Therefore, a profile analysis 
will be performed on the complex of parameters in relation to each dimension. Several tests am available in 
profile analysis. Of principal interest - for our purposes - is the "parallelism" test which help to answer the 
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question whether the profiles of two subpopulations aie parallel or not. If certain dimensions are helpful to 
detect non-parallel profiles, it might be interesting to know what parameter along the dimension does 
contribute most to ihese significant differences. Therefore, for non-parallel profiles, the analysis will be 
extended with a discriminant analysis (also called, the "level* test). At the theoretical level - as suggested 
in part 4.2 of this text - also the 'flatness' test might be relevant, since this test controls the similarity of 
responses for the different parameters along a dimension, independent of groups or subgroups. An answer 
to this question helps to support the validity of the different dimensions since the results indicate whether or 
not the dimensions/parameters are helpful to specify differences in the mastery of different components of 
the prior knowledge state. 

The grouping variable in our analysis is "University" (Ou- or RL-studeat). SPSS-PC+ MANOVA was used 
for the analysis. 

4.3.2 Control of underlying assumptions 

Profile analysis implies that some basic assumptions are met : 
Data screening revealed no missing data. 

Sample sizes are equal for both subpopulations (N— 91), so no special difficulties are 
expected. Moreover, only one independent variable is used. 

Attention has to be paid to a test of the homogeneity of the variance-covariance 
matrices. The evaluation of the homogeneity of variance-covariance matrices is based on 
the Cochrans and the Bartlett-Box F test. No problems are expected taking into account 
the equal sample sizes (91/91) and the satisfactory size of the samples. 
Mul tkol linearity is tested with the Bartlett test of sphericity. 

To evaluate assumptions about multivariate normality, boxplots of the mean submeasures 
for each dimension have been screened. In evaluating multivariate normality of the 
distribution of the mean % -scores, it is to be mentioned that the <r-values are very high, 
indicating a wide dispersion of the scores. This is to be expected, since the test measures 
" the prior knowledge state". 

Table 4 summarizes the data in relation to the evaluation of the homogeneity of variance-covariance 
matrices and the multicollinearity test. In relation to several dimensions, the assumption in relation to the 
homogeneity of the variance-covariance is violated ("subtopics" profile: micro-economics, "r jde relation": 
initiate, "behavioural level": apply, "content level": methods, "epistemological level": skills, 'representation 
level": symbolic representation). In our profile-analysis, this will be taken into account by using 
Greenhouse-Geisser and Huynh-Feldt adjustments to these univariate violations of homogeneity of variance- 
covariance. 

The Bartlett test of sphericity is significant in all cases, which means that the variables are highly 
intercorrelated. Although the p-values are very small, the SPSS-MANOVA-PC" procedure protects against 
instability caused by multicollinearity by excluding variables from the analysis with too low tolerance 
levels 6 . The fact, the MANOVA-procedure was never halted during execution indicates that 
multicollinearity did not cause problems. 



4 Tolerance ltvel » 1 - SMC (squared multiple correlation of each variable). 
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Table 4 : 

Analysis data in relation to muWcollinearity and homogeneity of variancc-covariance matrices. 





Homogeneity of Variance 
Cochran C Bardctt-Box F 


MuhicolUnearity 8 
Bartktt teat of sphericity f 


Uiijiili 


.61815 (p«. 023) 
.60009 (p». 056) 
.55177 <p«. 326) 
.joo/i {p— .iyij 
.57777 (p-. 139) 
.67642 (p-. 001) 
.53878 (p«. 462) 

.53235 (p«. 540) 


5.2052* 023) 
3.66013 <p«. 056) 
.96473 (p-J26) 

I. 70612 (p-. 192) 
2.19197 <p«. 139) 

II. 90118 001) 
.53878 <p«. 426) 

<710Q /•*_ jccm 

.37547 fe«.540) 


987.410 <p«.000) | 


| Utv tlx 
I Level2 


.53432 <p-. 4*1) 
.50285 (p«. 957) 


.4733* <p=. 4*2) 
.00290 <p=. 957) 


183.397 (p«.000) fl 


Quant 
General 


iOtU26>-.060) 
.54955 (p«. 347) 


3.536*5 <p«. 060) 
.88348 <p«.347) 


104.486 (p-.OOO) | 


Initiate 

Conseq 
I Manner 
y Property 

1 Know " 


.o2ool(p-.0U) 

.64513 (p-. 005) 
.50573 <p-. 914) 
.53602 (p-. 495) 
.57929 ft>- 131* 


5.*3145<p«.015) 
7.87839 (p«. 005) 
.01173 fe«.914) 

.46559 (p-. 495) 
i ^7aiw /wh ih\ 


487.276 (p«.000) 


1 Insight 
Apply 


.54883 (p«.264> 

.50539 <p«. 919) 
.69279 <p«. 000) 


1.24778 fe«.264) 

.01038 (p-. 919) 
14.40809 (p*. 000) 


390.814 (p~.000) 


Factual 

Concept 

Relat 

OiTUVl 

Method* 


.oloofl^p-.MSi — 

<ft<4ft 0171 
•jww tp™.yi 

.52711 (p-.608) 

.53813 (p«. 470) 

.65809 (p«. 002) 


4.00511 <p«.0J5) 
.UlO/o (p«.yl7) 
.26345 (p-. 608) 
.52216 (p«. 470) 

9.42804 002) 


644.228 (p«.000) 


TCoem ■ 

■ Kconcept 
A Episte 
| Logical 
Implem 


.5*050 k«.084) 
.52655 <p«.615) 
.51628 (p-. 758) 
.52689 (P-.611) 
.69798 (p«.000) 


2.*8114<p«.084) 
.25267 <p-. 615) 
0Q4QQ / nK 7c«\ 

.25927 <p« .611) 
15.26470 <p= .000) 


609.366 (p*= .000) 8 


Propo*2 
Propo*3 


.54853 (p«. 357) 
.53859 (p«. 465) 
.58282 (p*. 115) 


.84701 fe«J58) 
.53481 <p«.465) 
2.49016 <p«.115) 


459.793 (p-.OOO) 


"WSK 

Without 


JoJ07(p-.13S> " 

.52777 fe« .599) 


i. 5*005 fe«.2J*) 

.27649 599) 


239.921 (p-.OOO) 


Text 
Coocret 
Scheme 
Symbol 


.S3M1 <p-.45T) 

.50404 939) 
.56057 (p« .250) 
.67571 (p-.OOl) 


.55211 (p«. 458) 
.00584 939) 
1.32332 (p«. 250) 
11.79832 (p-. 001) 


414.124 (p-.OOO) 1 



9 

ERIC 



25 



Knowledge profiles of OU and RL students 



To evaluate multivariate normality box-plots of the mean % scores have been analyzed. Figure 13 presents 
e.g. a box-plot of the scores of Ou- and RL-students for the subtopic "marketing" on the subtopic-profile 
dimension* The * identifies the median and the box contains the middle 50% of the values* The lines 
emanating from the box extend to the smallest and largest observations in the subgroups that are less than 
one interquartile range from the end of the box. Points outside this range are marked with O (outliers) or 
even E (Extremes) if more than l.S interquartile distances away from the box. 
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Figure IS : Box-Plots 



Analysis of the box-plots for each variable in relation to each profile dimension reveal that there are outliers 
and extremes, but that their number remains restricted. Moreover, outliers are considered part of the 
particular distribution of our data* They are properly part of the population from which we intend to 
sample. Since the data reflect mastery of " the prior knowledge state" it is no surprise that the distribution 
of the data set does not reflect the "ideal" normal distribution. 

In general we can conclude that assumptions are met in order to execute a profile analysis on the research 
data available. Only in interpreting the analysis results in relation to the above mentioned categories we will 
have to take care. 
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4.3*3 Profile Analysis : Parallelism Test 

Table 5 presents an overview of the results for the parallelism test. This helps to answer the question 
whether the two different student groups have parallel or non-parallel profiles. This is commonly known as 
the test of parallelism and is the primary question addressed by profile analysis 7 . In relation to each profile 
dimension, Wilk's Lambda (X) was calculated and p-level determined. In the results table, Wilk's X is not 
reported in relation to four dimensions (marked with *). This is because these dimensions only contain two 
variables; in these cases a test of significance for Hoteling's T2 using the unique sums of squares was 
calculated, checking the interaction of the independent variable (student type) and the two dependent 
variables on the specific dimensions. 



Table 5 : 

Profile analysis date for the parallelism test 
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The data in table 5 are helpful to detect specific significant differences between Ou- and RL-students. The 
intermediate conclusion of non-significant differences between Ou-students and RL-students, based on 
analysis of the overall economics-score, can again be revisited by the refined breakdown of the profile 
analysis results. 

There are 7 knowledge profile dimensions which are helpful to illuminate significant differences between 
both student populations. 

4.3.4 Profile Analysis : Discriminant Analysis 

As mentioned in the introduction to our profile analysis a further analysis of non-parallel profiles can help to 
identify those parameters along the specific dimensions that contribute most to the differences between the 
two subpopulation (RL and Ou) 1 . In table 6, the results of this discriminant analysis are reported. Wilk's 
X can in this context be interpreted as the proportion of variability not explained by the group differences. 
In the fourth column of the table, we derived from this value the proportion of variability explained ((1-X) * 
100) by the group differences resulting from the independent variable "University". 

Table 6 : 



Results of the discriminant analysis 
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7 When usinj profile analysis as a substitute for univariate repeated measures ANOVA, the parallelism test is the test of interaction. 
9 Of course, the discriminant analysis is restricted to the non-parallel profiles. 
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The discriminant analysis can be extended by calculating structure coefficients? to determine the 
discriminatory power of the separate parameters along a knowledge profile dimension. The results of this 
analysis are found in table 7. 



TabU7: 
Overview of structure coefficients 
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A structure coefficient indicates the correlation between a parameter and the discriminant function. High 
values indicate important discriminant effects. Table 7 demonstrates that some parameters have structure 
coefficients up to > .5 . If we combine these results with the dimensions relevant to a statistical significant 
degree to differentiate between populations, we can conclude that the two university populations can 
especially be differentiated along the following two knowledge profile dimensions : the content level 
dimension and the epistemologicai level dimension. 

4.3.S Profile Analysis Results : Flatness Test 

Is the mastery of the prior knowledge state as defined by the parameters along a dimension different, 
independent of the groups (a within-subjects main effect) ? In other words, do students master the prior 
knowledge state in a similar way as defined by the different parameters along a dimension ? This question 
is especially relevant for parallel profiles, since in non-parallel profiles at least one parameter is not flat; 
nevertheless also the results in relation to non-parallel profiles are reported. 



9 Since the subvalues on each profile dimension are highly intcrcorrelated, we cannot use raw or standardized discriminant function 
coefficients. The highly correlated variables 'share* the diicrirntnants weights. It is safer to base our interpretation on the structure 
coefficients whfah sre less likely to be influenced by these intercorrstations. 
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Table 8 : 

RcaulU of the flatness tot in profile analysis 
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If the flatness test is non-significant, then the profiles are not helpful to clarify or detect differences in the 
mastery of different components of the prior knowledge state. The results of the flatness test are therefore 
also of relevance to determine the validity of the knowledge profile dimensions. 

The results of the flatness test are found in table 8. For each dimension Wilk's X has been calculated, with 
the exception of the three dimension where only two parameters are available along the dimension; there the 
F-value is reported (marked with *). 

All dimensions result in non-flat knowledge profiles. This implies that all dimensions are helpful to identify 
a specific structure in the mastery of the prior knowledge state. Following this structure, the mastery of 
certain components 10 of the prior knowledge state is better than for other components. 



5 Conclusions 

In this text, we attempted to analyze the prior knowledge state of two specific university subpopulations, 
studying a multifunctional course. In the present study, the overall economics score was not significantly 
different between both university populations. But in this research, special attention was paid to a further 
elaboration of this general economics-score by grouping items along a variety of knowledge profile 
dimensions. After outlining - in short - the theoretical base of these dimensions, we focused on the 
application of these dimension to compare the knowledge profiles of the two university populations. A 
profile analysis (parallelism test and discriminant analysis) could help to reveal specific significant 
differences between the profiles of both student populations. Up to seven of the ten knowledge profile 
dimensions proved to be of relevance. Especially the "content dimension" and the "epistemological 
dimension" were helpful to describe these differences. 

A further extension of our profile analysis (flatness test) helped to induce further evidence to support the 
validity of the knowledge profile dimensions. 

The present study is of importance since we succeeded in defining and operationalising a new and promising 
approach towards the analysis of the prior knowledge state. It is foreseen that in situations where there are 
significant differences between the prior knowledge state of specific subpopulations, the profile dimension 
are helpful to detect and dissect the strengths and weaknesses of the students involved. This might be a 
promising starting point for differentiated diagnostic and guidance approaches. 



w The concept •components" refers to this subpart of the prior knowledge state that can be isolated in connection to a specific 
parameter along a knowledge profile dimension. 
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